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0 BACKGROUND - FIELD OF INVENTION 

This invention relates to methods of estimating the subjective quality of a multimedia 
communications system or distributed applications software system in which the quality of the 
received audio, voice, video or the response time of the software system may be degraded by time 
varying impairments and in which it is accordingly desirable to have some means of measuring or 
estimating the subjective or perceptual quality of the decoded audio, voice, video or software 
system. 



It is desirable to have some means of measuring the quality of a distributed system in order that it 
can be verified that the system is operating correctly and that user expectations are being met. In the 
case of a speech transmission system such as a telephone network this measurement is sometimes 
made subjectively by using a test group of people or test surveys. The results of a subjective voice 



quality test are usually given as a Mean Opinion Score (MOS). As it is generally impractical to 
conduct subjective tests during the normal operation of a speech transmission system objective 
testing methods have been developed. These use measured statistics related to the speech 
transmission system to predict a subjective performance level and typically determine an R rating 
factor. 

Modern communications systems often used some form of shared transmission medium, for 
example wireless, local or wide area packet switched network. This increases the prevalence of time 
varying transmission impairments unlike the more constant and predictable impairments 
characteristic of older circuit switched networks. 

It has been found during subjective testing that bursts of noise or distortion occurring late in a call 
have greater effect than the same bursts occurring early in the call, as illustrated in Figure 3. This 
effect is thought to be due to the operation of human short-term memory which retains information 
on the most recent events and remembers detailed information on heard events for typically two 
minutes. If a series of bursts of noise or distortion occur during a telephone conversation then 
according to the known and accepted psychological theories of short term memory and recency the 
user would tend to remember the last burst and have a general recollection of the call quality up to 
that point. Experiments by Boneau and Daily (Psychological Association of America, 1995) 
indicate that the short-term memory loss process is exponential in nature. 

Current systems for estimating voice quality do not consider the effects of the temporal location of 
impairments on the user and hence do not give an accurate representation of subjective voice 
quality. The present invention relates to methods for estimating subjective voice quality that 
consider temporal effects and hence can provide greater accuracy. 

Distributed applications software can exhibit similar problems. Network or server loading may vary 
with time, which results in a user experiencing variable response time. It is expected that detailed 
subjective testing of user satisfaction would reveal similar characteristics to those found in testing 
telephone networks as the same basic memory processes are at work. The present invention 
provides a new approach for estimating the subjective quality of distributed applications software. 



BACKGROUND - DESCRIPTION OF PRIOR ART 



Prior art systems for measuring voice quality, as described by Douskalis (Hewlett Packard 2000), 
Royer (US 5,710,791) and Di Pietro (US5,867,813), use centralized test equipment which samples 
the voice quality from various conversion points. A loop back condition is established at an 
conversion point, the test equipment transmits a known signal and then compares the received 
(looped back) signal with the original, thereby estimating delay, distortion and other impairments. 
This approach provides an accurate measure of voice distortion but only provides this measure for a 
sample conversion point and under the network conditions that existed at the time of the test. No 
provision is made in these systems for estimating the effects of the temporal location of impairments 
on quality. 

Another approach currently used for estimating voice quality is to estimate the subjective 
performance of the voice connection using objectively measured parameters. Models such as the E- 
Model described by Johannesson, (IEEE Communications Magazine 1997), are able to produce R 
ratings which can be correlated to user perceived voice quality. This process is applied by a central 
management system which gathers statistics on noise and delay and then produces an estimate of 
voice quality. This method as described by Johannesson does not consider the effects of the 
temporal location of impairments. 

Rosenbluth (Tl committee contribution T1A1. 7/98-031) described the result of a subjective voice 
quality test conducted by AT&T. This test employed a set of 60 second voice samples which were 
corrupted at the beginning, middle and end. Rosenbluth reported that the MOS score typically 
ranged from 3.8 for a sample corrupted at the beginning to 3.2 for a sample corrupted at the end. 
Rosenbluth proposed a mathematical model for determining the estimated voice quality of the 
speech sample which comprised the steps of dividing the sample into segments, computing a MOS 
score for each segment, computing a weight for each segment that was a function of the proportional 
position of the segment within the overall sample and then computing the sum of the segment MOS 
scores multiplied by their respective weights. This approach suffers from several major drawbacks: 



(i) Rosenbluth determined the weight for each impairment based on the proportional location of 
the impairment within the speech segment - i.e. determined a number between 0 and 1 that 
represented the position of the impairment between the start and the end of the segment. 
This is counter-intuitive as it would give the same weight to an impairment occurring 30 
seconds before the end of a one minute call to an impairment occurring 30 minutes before 
the end of a one hour call whereas the process by which information is lost from human 
memory is time dependant. 

(ii) The equations used by Rosenbluth were computationally complex, which would be 
acceptable in an off-line determination of voice quality based on recorded data but not for 
real-time computation. 

(iii) Rosenbluth proposed a testing methodology in which a series of short test messages of 
known time duration could be sent and then the results from each test message combined. 
Rosenbluth did not consider a test methodology in which quality was continually estimated 
during the normal operation of the system. 

Accordingly, there is a need for a means of estimating subjective quality in a voice communications 
system that considers the temporal location of impairments and is computationally efficient. 

Furthermore, there is a need for a means of estimating subjective quality in a voice communications 
system that considers the temporal location of impairments and is able to continuously monitor 
voice quality during the normal operation of the communications system. 

Furthermore, there is a need for a means of estimating subjective quality in a video transmission 
system that considers the temporal location of impairments and is computationally efficient. 

Furthermore, there is a need for a means of estimating subjective quality in an audio transmission 
system that considers the temporal location of impairments and is computationally efficient. 



Furthermore, there is a need for a means of estimating subjective quality in a distributed applications 
software or client-server system that considers the temporal location of impairments and is 
computationally efficient. 

SUMMARY 

The present invention provides an improved means of estimating subjective quality in a 
communications systems wherein the temporal location of impairments within the data stream are 
considered and the subjective effects of said impairment determined based on the time delay 
between the occurrence of said impairment and a time reference at which the quality estimate is 
being determined. 

Objects and advantages 

Accordingly, besides the objects and advantages of the present invention described above, several 
objects and advantages of the present invention are: 

(a) to provide a method for estimating the subjective or perceptual quality of a multimedia 
communications system which considers the temporal location of impairments 

(b) to provide a computationally efficient method for estimating the subjective or perceptual quality 
of a multimedia communications system 

(c) to provide a computationally efficient method for estimating the subjective or perceptual quality 
of a distributed applications software system 

Further objects and advantages are to provide a method of estimating subjective quality for an audio 
or video streaming system in which packetized audio or video is broadcast or multicast through a 
packet network and for a multimedia conferencing system. Still further objects and advantages will 
become apparent from consideration of the ensuing description and drawings. 



DRAWING FIGURES 

Embodiments of the invention (the Quality Monitor) are described below by way of example only 
with reference to Figs. 1 to 5 of the accompanying drawings, of which:- 

Fig 1 shows an example of the use of the Quality Monitor in a multimedia communications system 

Fig. 2 shows an example of the use of the Quality Monitor in a distributed software applications 
system 

Fig. 3 shows an example of the effects of temporal location on subjective voice quality 

Fig. 4 shows the architecture of the Quality Monitor 

Fig. 5 shows a flowchart for the Quality Monitor 

Fig. 6 shows an example of the operation of the Quality Monitor. 

DESCRIPTION 
Preferred embodiment 

Fig. 1 shows an example of a multimedia communications system. Audio or video source (6) 
transmits a stream of audio or video data to user interface (5) by means of transmitter (1), 
transmission network (2) and receiver (3). Transmission network (2) is subject to time varying 
impairments which may include jitter, noise, distortion, variations in amplitude and other such 
impairments. Quality Monitor (7) is connected to or contained within Receiver (3), monitors the 
received data stream and reports the estimated subjective quality to management system (8). 

Fig. 2 shows an example of a distributed applications software system. A user interacts with a 
remote Server (10) by means of User Interface (13), Client Software (12) and Transmission Network 
(1 1). Requests from the user for data are submitted via User Interface (13) to Client Software (12). 
Client Software (12) forwards a representation of the user request through Transmission Network 



(1 1) to Server (10). Server (10) sends a response through Transmission Network (11) to Client 
Software (12). Client Software (12) communicates the response to the user by means of User 
Interface (13). The response time between the user request and the response may vary depending on 
the loading and capacity of Transmission Network (11) and Server (10). Quality Monitor (7) is 
connected to or contained within Client Software (12), monitors the transmitted and received data 
stream and reports the estimated subjective quality to management system (8). 

Fig. 3 shows some results (from Rosenbluth) of the effects of temporal location on subjective 
quality for a voice communications system, by way of illustration to support understanding of the 
following description. It can be seen that for an impairment occurring early in the call the subjective 
voice quality was 3.8, which is quite good quality, whereas the identical impairment occurring at the 
end of the call resulted in a subjective quality of 3.2, which is considerably lower quality. 

Fig 4. shows the architecture of Quality Monitor (8) as employed in a voice communications system. 
The structure of Quality Monitor (8) for other applications would be similar however the 
terminology used for impairments may vary. 

Impairment Measurement functions (20), (21), (22) are connected to an Instantaneous Voice Quality 
Rj function (23) and measure the value of transmission system impairments during a time interval. 
In the preferred embodiment Impairment Measurement function (20) measures end to end 
transmission delay, Impairment Measurement function (21) measures packet loss percentage and 
Impairment Measurement function (22) measures the quality of the base voice coding algorithm. 
Instantaneous Voice Quality function (23) is connected to one input of a Comparator (24), an 
Increment function (27) and a Decrement function (26). The inputs of R Counter (25) are 
connected to Increment function (27) and Decrement function (26). The outputs of R Counter (25) 
are connected to Comparator (24), Average function (28), Minimum function (29), Maximum 
function (30) and to Output to Management System function (31). Average function (28) is fed back 
to Increment function (27). 

Fig. 5 shows a flowchart of the operation of Quality Monitor (8) as employed in a voice 
communications system. 
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Fig. 6 shows an example of the output of Quality Monitor (8) as employed in a voice 
communications system. 

Description of Operation 

The following sequence of operations is executed at periodic intervals. In the preferred embodiment 
said periodic interval is one second however shorter or longer times may be used. A transmission 
quality factor R is determined using the procedures described in detail below. Subjective voice 
quality is related to quality factor R according to the relationship: 





R 


Subjective voice qualitv 


90 


- 100 


Very satisfactory/ excellent 


80 


-90 


Satistfactory/ good 


70 


-80 


Some users dissatisfied/ fair 


60 


-70 


Many users dissatistfied / poor 


50 


-60 


Most users dissatisfied 


0 


-50 


Unacceptable 



Impairment measurement function 1 (20) determines the base voice quality factor for a particular 
voice coding algorithm for Rc . Flowchart step Determine Base Quality (40) comprises the process 
of identifying which voice coding algorithm is being used and then using a predetermined table of 
values to determine the value Rc. A typical initial Rc value for a voice coding algorithm is 84. 

Impairment measurement function 2 (21) determines the quality reduction due to the end to end 
transmission delay. Flowchart step Determine Quality Degradation (41) comprises the process of 
using the measured end to end delay to determine the delay quality degradation factor R<j. In the 
preferred embodiment this is determined using the piecewise linear approximation: 

If delay < 1 75mS then Flowchart Step 41 

R<i = delay * a 

else 

R<i = (delay - b) * c 



In the preferred embodiment delay is the one way transmission delay plus the delay due to the voice 
compression process, a = 0.02, b = 147 and c = 0.125 however other values may be used. 

Impairment measurement function 3 (22) determines the quality reduction due to the percentage of 
voice packets lost since the last measurement. Flowchart step Determine Quality Degradation (42) 
comprises the process of using the measured packet loss to determine the packet loss quality 
degradation R p . In the preferred embodiment this is determined using the piecewise linear 
approximation: 

If loss < 5% then Flowchart step 42 

R p = loss * ei 

else 

Rp = loss*e2 + f * (ei - e2) 

In the preferred embodiment loss is the percentage of packets lost since the last measurement, ei = 4, 
e2 = 2.5 and f = 5 however other values may be used. Preferably the values of ei and e2 are obtained 
from a table which holds specific values for each voice coding algorithm. 

The instantaneous voice quality factor Rj (23) is then determined by subtracting the reduction factors 
(21) and (22) from the base R value (20) for the voice coding algorithm. Flowchart step Determine 
Voice Quality Factor (43) comprises the process of using the previously determined quality factors 
to determine the value of Ri (23): 

Ri = Rc - R<j - Rp 

Comparator (24 and flowchart step 44) is used to determine if the value of Counter (25), which 
represents the voice quality factor R, should be increased or decreased. If Ri (23) is low then the 
user is experiencing poor quality and voice quality factor R (25) is decreased rapidly. If Rj (23) is 
high then the user is experiencing good quality and voice quality factor R (25) is increased 
exponentially towards the average voice quality factor Ra V (28). 



IfR<Ri then 



flowchart step 44 
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R = R + (Rav-R)/g 



flowchart step 45 



else 



R = R - h 



flowchart step 46 



In the preferred embodiment g = 200 and h = 4 however other values may be used. 

Flowchart step 47 updates the average (28), minimum (29) and maximum (30) values of R (25). 
The average voice quality factor Ra V (28) is determined by adding the value Rj to a sum R sum , 
incrementing a count of the number of values and dividing R sum by said count: 



The maximum and minimum values of R are determined by setting R^ (30) equal to R (25) if R 
ever exceeds Rmax and by setting R,™ (29) equal to R if R is ever below R min . 

The parameters R (25), Ra V (28), R m i n (29) and R^x (30) are updated according to the procedures 
described above at regular intervals, preferably every second, and the values may be read at any 
time by a central management system through Output to Management System (31) or may be 
incorporated into an end-of-call record for later analysis. 

Alternative embodiments 

The subjective effects of additional impairments such as jitter, clipping, noise and amplitude 
limiting may be determined and subtracted from the value of Ri using the same basic procedures as 
described above. 



R, 



sum 



N = N+ 1 




The value of Ra V may be determined either by averaging the output value R or the instantaneous 
value Rj. 
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The periodically obtained sets of measured impairment values may be stored and at some later time 
retrieved for analysis using the same procedure described above. 

An alternative means of determining a quality factor that represents the temporal position of 
impairments is to measure the impairment values at regular intervals, at each such interval to 
compute an estimated subjective quality value, and to compute a weighted sum of each subjective 
quality value multiplied by a function of the time difference between the time at which the 
impairment measurement was made and the time reference for which the subjective quality is being 
determined. 

The present invention may be used to measure the subjective quality of a distributed applications 
system, as shown in Figure 2. The measured impairments would include signal reception data rate 
and response time, both of which may be affected either by network bandwidth or server loading. 



T : Conclusion, Ramifications and Scope 



One benefit of the present invention is that the subjective speech quality of a voice communications 
system may be determined more accurately as the effects of the temporal location of impairments 
are considered. 



fa Another benefit of the present invention is that the computational complexity is low and hence the 
quality monitoring system can be implemented at low cost. 

Although the description above contains many specifities these should not be construed as limiting 
the scope of the invention but as merely providing illustrations of some of the presently preferred 
embodiments of this invention. For example, the present invention could be used for monitoring the 
quality of a cellular telephone system, a digital broadcast video system or the user interface on a 
software application. 

Thus the scope of the invention should be determined by the appended claims and their legal 
equivalents, rather than by the examples given. 



